ITEP-TH-48/09 



Information Theory and Renormalization Group Flows 
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We present a possible approach to the study of the renormalization group (RG) flow based en- 
tirely on the information theory. The average information loss under a single step of Wilsonian RG 
transformation is evaluated as a conditional entropy of the fast variables, which are integrated out, 
when the slow ones are held fixed. Its positivity results in the monotonic decrease of the informa- 
tional entropy under renormalization. This, however, does not necessarily imply the irreversibility 
of the RG fiow, because entropy is an extensive quantity and explicitly depends on the total number 
of degrees of freedom, which is reduced. Only some size-independent additive part of the entropy 
could possibly provide the required Lyapunov function. We also introduce a mutual information 
of fast and slow variables as probably a more adequate quantity to represent the changes in the 
system under renormalization and evaluate it for some simple systems. It is shown that for certain 
real space decimation transformations the positivity of the mutual information directly leads to the 
monotonic growth of the entropy per lattice site along the RG flow and hence to its irreversibility. 



I. INTRODUCTION 

Renormalization group (RG), which is a powerful in- 
strument for analyzing different strongly coupled systems 
[l|(see also 2] for more recent reviews), is usually based 
upon an appropriate division of the whole set of variables 
into two subsets: so called "fast" and "slow" variables 
with subsequent elimination of the fast ones. The result- 
ing slow system is usually expected to behave essentially 
in the same way as the initial one but with coupling con- 
stants changed. Successive application of this transfor- 
mation results in a flow in the space of couplings. Af- 
ter each step of the RG transformation the information 
about exact values of fast variables is lost for the observer 
to whom only slow variables are available. It seems nat- 
ural then to expect, that this information loss should 
lead to the irreversible character of the corresponding 
RG flow. 

This irreversibility is a subject of an intensive study 
for several decades now, starting from the famous work 
of Zamolodchikov Q , who showed under certain assump- 
tions that in two dimensions (2D) there exists a function 
on a space of coupling constants (a kind of Lyapunov 
function) which always decreases along the RG flow and 
at fixed points coincides with the central charge c of 
corresponding conformal theories. "Irreversibility" here 
means that the flow in the space of couplings resembles 
that of a simple dissipative system with a given trajec- 
tory never returning back to its starting point, thus ex- 
cluding limit cycles or more complex strange attractors. 
In 2D theories the c-theorem of Ref. [3] states that the 
RG evolution is exactly of this type and looks like a sim- 
ple monotonic flow downhill in the couplings' space from 
fixed points with large c to those with smaller values of 
the central charge. 

Much work have been done in this direction and the 
c-theorem was studied in detail with its possible general- 



* Electronic address: ' apenko®lpi.ru| 



izations to higher dimensions (see e.g. Q and references 
therein and also [11 for quite recent development). Still 
the interesting possibility that in general the flow may 
be more complicated, and may even exhibit chaotic be- 
havior, like in case of many other nonlinear mappings, 
attracts some attention [^ (probably the first example of 
a chaotic RG fiow was found for spin systems on hierar- 
chical lattices |l7|], see also [1,[§|). The models where such 
peculiarities are observed are however rather artificial, so 
that the question whether they could take place in more 
realistic cases remains somewhat unclear. 

Though this problem is rather complicated mathemat- 
ically it looks much simpler from the physical point of 
view. The Wilsonian RG transformation is designed 
in such a way that all finite dimensionless correlation 
lengths in a translationally invariant system always de- 
crease [l| . To say this another way, all masses (measured 
in units of an ultra-violet cut-off) grow and it seems quite 
natural that the effective number of massless modes, (in 
2D measured by the central charge c) cannot increase. 
The argument based on the growth of masses, though 
not completely rigorous, is quite general, hence some- 
thing unusual, like limit cycles in the flow, may be ex- 
pected only in some special systems with only infinite 
correlation lengths or with infinite number of correlation 
lengths which can be arbitrarily large (when we have e.g. 
a self-similar spectrum of masses converging to zero [lo|). 
But in systems with finite number of well defined corre- 
lation lengths the RG flow should probably always be 
irreversible. 

This argument however seems to be unrelated to the 
aforementioned information loss and information aspects 
probably play no role in the irreversibility, which may 
seem somewhat strange. Probably the most known at- 
tempt to study RG irreversibility using information the- 
ory tools is the approach of Ref. [ll|, where the rela- 
tive entropy was introduced in quantum field theory and 
some monotonicity theorems for it, as a function of rele- 
vant couplings, were proved and its relation to c-theorem 
was discussed. 

In this paper we present a somewhat different, entirely 
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information theory based approach to the problem of 
the RG flow, oriented more on discrete real space RG 
transformations in different lattice spin systems. Clearly, 
information-theoretic considerations are too general to 
lead to some non-trivial concrete results, but still, as we 
shall see below they may impose some restrictions on the 
possible character of the flow. First, in Section 2 we eval- 
uate an exact amount of information loss for a single step 
of the renormalization as a kind of conditional entropy of 
fast variables (compare with [13|). As one might expect, 
this information loss is equal to the decrease of entropy 
under the RG transformation. Thus it is possible to es- 
tablish a general theorem about the monotonic decrease 
of the total entropy (and relative entropy as well) under 
general coarse graining as a result of information loss. 

The monotonic decrease of entropy along the RG flow 
is neither something unusual (see e.g. nor very inter- 
esting, because the entropy is proportional to the total 
number of degrees of freedom and should certainly de- 
crease when some of them are integrated out. Hence this 
monotonic behavior does not directly lead to the irre- 
versibility of the flow. An interesting thing may happen 
however, if the entropy of a finite system possesses an 
additive size-independent part. If this part of the en- 
tropy also decreases, as the total entropy does, then it 
may provide the proper Lyapunov function, because it 
does not explicitly depend on the number of variables 
and its change is entirely due to the flow of coupling con- 
stants. We will show that this takes place in ID Ising 
model, where this subextensive part of the entropy is 
known as the "excess entropy" and was studied in detail 
as a measure of statistical complexity of spatial structures 
[T5| . This excess entropy equals to the mutual entropy 
of two halves of the system which is actually the reason 
why it decreases under coarse graining. Unfortunately 
we are unable to present a general theorem concerning 
the monotonicity of such excess entropy in higher dimen- 
sions. 

It appears possible though to prove quite a different 
monotonicity theorem, but only for a certain class of 
real space lattice RG transformations, namely for dec- 
imations, when exactly half of variables are eliminated 
on each step. The proof is based on the analysis of a new 
quantity introduced in Section 3, — mutual information 
of slow and fast variables. Contrary to the information 
loss this mutual information shows how much informa- 
tion about eliminated fast variables is still present after 
a single step of the Wilsonian renormalization. The non- 
negativity of the mutual information may alone impose 
some restrictions on the character of the flow leading to 
its irreversibility. 

For the decimations of the aforementioned type in lat- 
tice models we will show that the entropy per lattice 
site monotonically grows as a result of the positivity of 
the mutual information of spins on two identical sub- 
lattices, one of which is eliminated in the course of the 
RG transformation. Therefore such decimation transfor- 
mations, though they results in highly non-linear map- 



pings, always lead to irreversible flows in the space of 
couplings. Simple examples of such a behavior are dis- 
cussed in Section 4, where one- and two-dimensional clas- 
sical spin models and also the continuum limit Gaussian 
model are considered. Note, however, that the theorem 
holds true only for systems with real actions, when infor- 
mation theory can be applied, while e.g. it is not valid 
for the Ising model with complex external magnetic field 
where the decimation RG flow is known to be chaotic [9| . 
The theorem is also invalid for spin models on hierarchi- 
cal lattices 0, because decimations are not associated 
there with decomposition of these finite and inhomoge- 
neous lattices into two identical sublattices. 

The entropy per site growth may seem somewhat un- 
expected, because it takes place not only in the trivial 
case, when the flow goes to the high-temperature fixed 
point, but even when we start from points located in the 
ordered phase. RG trajectories that start from points 
below the phase transition can never end in the fully dis- 
ordered trivial fixed point, since for decimation the mean 
magnetization is conserved along the flow, but neverthe- 
less the entropy per site (and hence the amount of dis- 
order) will always grow. This is a peculiar feature of 
the decimation and does not take place for other possi- 
ble renormalization schemes, like e.g. the majority rule 
block spin transformation also discussed in Section 4. 



II. INFORMATION LOSS AND THE RG FLOW 

Suppose that we somehow manage to divide our vari- 
ables into "fast" -0 and "slow" ones rj. For the Wilsonian 
momentum space RG the fast variables are high Fourier 
harmonics of the fields, but for real space renormalization 
in lattice systems i/i's are just spins on some sublattice, 
so that they are not actually "fast" in any sense. Still 
we will use these names to distinguish degrees of freedom 
to be integrated out and those that remain. The parti- 
tion function is given by the sum over all fields, which 
symbolically may be written as 

Z = ^exp(-5(0,r7)), (1) 

•tp,ri 

where S{'ip, rf) is the action (or energy divided by temper- 
ature for classical lattice systems). The fields normally 
depend on all space-time coordinates, so that the sums 
in Eq. ([1]) are actually path integrals, properly regular- 
ized. Now, the first step of the RG transformation is to 
integrate out the fast variables "0 to obtain an effective 
action for the slow ones, S'{ri), defined by 

exp(-^'(r7)) = ^exp(-^(0,77)) (2) 

•v. 

This transformation is then repeated many times result- 
ing in the RG flow of the corresponding actions (or cou- 
pling constants if the form of the action is fixed). 
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A. Information loss and entropy 

Certainly, after the fast variables are eliminated, in- 
formation about their exact values is lost. One can ask 
then, what is the precise amount of the information loss. 
For a moment let us fix the slow variables 77, then the 
conditional probability for ip (normalized to unity) is 

p(V^|7?)=exp(-5(V',77) + 5'(7?)), (3) 

E0P(V'I?7) = 1- 

It seems natural then to define the information (measured 
in 'nats') related to some field configuration -ip in the fixed 
background 77 by the usual Shannon formula 

/0 = -lnp(V'|77), (4) 

and the information loss after the fields ^ are integrated 
out as an average of ([4]) over the distribution ([3]) 

This quantity still depends on the chosen configuration 
of the slow fields 77. Actually (J^) is similar, in a sense, to 
the Boltzmann entropy of a 'macrostate' specified by the 
coarse-grained variables rj. Indeed, in case of 'equiparti- 
tion', when p(V'|'?) does not depend on ^ in some domain 
of fast variables the information loss Eq.(l5]) is just the 
logarithm of the number of 'microstates' corresponding 
to a given macrostate. 

We propose then that the total information loss SI is 
an average of (J^) over all configurations of rj with the 
corresponding weight p{ri) — 1/Zexp(— S"), i.e. 

(5/ = -^p(77)^p(7/;|77)lnp(V'|?7). (6) 

This is obviously the conditional entropy H{ip\ri) of two 
sets of random variables ip and 77 well known from the 
information theory [l6l |. This is the average entropy of 
the fast variables given the state of the slow subsystem. 
The conditional entropy of this kind usually measures the 
amount of information loss in a noisy channel when ip is 
an input signal and 77 is an output. One can also view 
the RG transformation as an action of a smoothing filter, 
then SI is the information loss due to smoothing. The 
conditional entropy as a measure of information loss also 
have been introduced in Ref. [l^l where a coarse graining 
of polymer configurations was considered. 

The above estimate of the information loss introduces 
a somewhat new paradigm of renormalization. We see 
now that it is possible to view the fast fields ip as an in- 
put signal, not directly available to a receiver, while the 
slow field 7] is an output. After renormalization only the 
output signal is available and one can only try to restore 
the input signal more or less accurately. This is a typical 
situation of a signal transmission through a noisy channel 
where information loss may be estimated as a conditional 



entropy ([6]). But this analogy suggests that it may prob- 
ably be of more use to discuss not only the information 
loss but also the information that is preserved, i.e. infor- 
mation about fast variables that is stored in the action 
for the slow ones (see Section 3). 

Substituting the probability distributions into Eq. ([6]) 
we easily obtain 

SI = H{i/j\r^)^{S)s-{S'}s'=H-H', (7) 

where the brackets (. . .) mean the average over the fields 
with the corresponding actions (clearly, for continuous 
theory some regularization is required for these formulas 
to make sense), the initial informational entropy H of the 
distribution piip, 77) is given by 

H = - ^p(7/;, 77) \np{ip, r,) ^{S)s + In Z, (8) 

where 

P{V, '^) = ^ exp(-S'(?7, V')) (9) 

and the entropy of the slow field H' = II{r}) is given 
by the same formula with ^(77, ip) — >■ ^(77) and S S' . 
Recall that our RG transformation is defined in such a 
way that the partition function Z remains unchanged. 
Also note that the new action S", defined by ([2]), always 
contains an additive constant term, but it does not con- 
tribute to the entropy and may be ignored in the evalu- 
ation of H' . 

The Shannon entropy H used here is the entropy of the 
fluctuating fields and it measures in fact how large these 
fluctuations are. For a classical system H coincides with 
its thermodynamic entropy iJ^, but in quantum systems 
(in the continuum limit) they are different, because Hj- 
is related to fluctuations in occupation of energy levels 
and not to that of fields as functions of space-time coordi- 
nates. While H is dominated by an ultra-violet divergent 
"bulk" contribution, proportional to the space-time vol- 
ume, the quantum thermodynamic entropy Ht looks, in 
a sense, as a finite-size correction to a leading term in H . 
Indeed, for example, for a 2D system on a large cylinder 
of length L and circumference /3 which may be viewed as 
a ID quantum system at a temperature T = 1//3 we have 
hiZ Lf3 and the same is true for H , but this bulk term 
obviously does not contribute to the thermodynamic en- 
tropy Ht ~ d{T\nZ)/dT . Hence non-zero Ht arises 
only from finite size corrections to In Z (see Section 4.5 
for another example). 

Thus the information loss SI is merely the difference of 
the initial entropy H = H{ip,ri) and the entropy of the 
remaining slow variables H{'q). For discrete variables 7/, 
i/" (e.g. for classical spin systems) both entropies and 
the conditional entropy are always nonnegative, H > 0, 
i/' > and i/(V'|7?) > and hence 

H > H', (10) 
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which means that the entropy of the remaining variables 
decreases along the RG flow. 

At first glance this decrease of entropy may seem 
strange, since if the flow is e.g. directed toward the high 
temperature fixed point the system obviously gets more 
disordered after renormalization. Recall, however, that 
it is the total entropy, an extensive quantity proportional 
to the system size, that decreases according to Eq. (fTO|) 
and this decrease is mainly due to the fact that the to- 
tal number of variables is reduced after some of them 
are integrated out. For example, in large lattice sys- 
tems H ~ Nh, where N is the number of sites which 
decreases under real space RG transformation. The en- 
tropy per lattice site h which is a more adequate measure 
of disorder may well increase along the RG flow. 

Therefore the information loss 51 is not very informa- 
tive about the change in physics under renormalization 
and the monotonic decrease of H does not in general give 
rise to a proper Lyapunov function defined on the space 
of coupling constants. 

This is not very surprising because the total infor- 
mation loss is a rather crude characteristic of the RG 
transformation. An interesting possibility still remains, 
that some part of this information loss may provide the 
required monotonic function. The most natural choice 
probably would be some volume-independent additive 
part in H for a finite system, something that may be 
called e.g. information about boundary conditions. An 
example of such a function will be explicitly presented in 
section 4.1 for the one-dimensional Ising model. How to 
obtain possible extension of this result to higher dimen- 
sions is however not clear. 

To conclude this subsection it should be mentioned, 
that for continuous field variables when we deal not with 
true probabilities but probability distributions and sums 
are replaced by integrals, direct application of the above 
formulas may lead to negative values of entropies [l^ 
and to negative 51 which certainly is not acceptable. To 
avoid this one should either somehow make the variables 
ry, V' discrete ("digitize" them) or better use some other 
quantities, which do not suffer from such a drawback, like 
e.g. the relative entropy. 



these probability distributions 



Hrel = D{^l\\u)=Y,^M) In 



(11) 



This distance, though not symmetric, is nonnegative even 
for continuous (j) f^nd Hrei ~ for fj, — v. 

Now, the main idea of the approach [ll| is to consider 
probability distributions corresponding to two actions S'l 
and 5*2 which differ by the values of coupling constants, 
and to study the relative entropy Hr^i = D{pi\\p2) where 
Pi ~ exp(— S'l) and p2 ~ exp(— 52), and then choose p2 
to correspond to a RG fixed point distribution. Thus 
physically this Hrei measures how far our system with 
the action S'l is from a given fixed point. Then we expect 
the relative entropy to behave monotonically under the 
change of the coupling constant, corresponding e.g. to 
some relevant perturbation of the fixed point. 

In fact, some rather general results are known about 
the behavior of Hrei under a coarse-graining. Let us pro- 
ceed as usual and decompose the whole set of fields (p 
into fast ip and slow ones rj. If we introduce conditional 
probability ^(-0177) (normalized to unity J2'4,Pi^\v) = 1) 
and make use oi p{ip,ri) = p{Ti)p{ip\ri) we can write the 
following chain of equalities 



Pi(V',»7) 



^Pi{v)Pi{i^\v) In 
'/',»? 

Piiv) _ 

P2{V) 



P2{'ip,v) 

piiv^piii'lv) 

P2{v)P2{ll'\v) 



pi(V'l^) 
P2{il^\v) ' 



(12) 



The first term in the resulting expression is obviously the 
relative entropy after renormalization, iJ^gj, while the 
last term is known in information theory as a conditional 
relative entropy which is always nonnegative . Hence 
we have 



(13) 



B. Relative entropy 

Relative entropy is the basic concept of the approach 
to renormalization developed in a series of papers (TTI . 
[T^ . For the sake of completeness we present here a short 
discussion of its properties with regard to the Wilsonian 
RG transformation. 

Consider some statistical system with variables 4>, and 
two probability distributions ^i{4>) and ^{(j)). Then we 
may introduce the relative entropy Hrei as the so called 
KuUback-Leibler distance (divergence) D{fj,\\i/) between 



This result, as well as the derivation of Eq. is in 

fact widely known and usually referred to as a decrease 
of relative entropy under any kind of coarse-graining [l^ . 
The decrease of relative entropy for continuum Gaussian 
model with lowering the ultra-violet cutoff ^11,] is just one 
example of this general phenomenon. 

One can also view Eq. ([T^ as a generalization of the 
expression ([7]) for the information loss. Then the last 
term in Eq. (|12l) may probably be called "conditional 
information loss" . Since this is nonnegative also for con- 
tinuous variables it looks more preferable for quantum 
field theory, than the information loss from the previous 
subsection. 
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Unfortunately inequality p3|) for the relative entropy 
is also not very useful because normally Hj.ei is an exten- 
sive quantity and its monotonic decrease again may be 
attributed to the reduction in the number of degrees of 
freedom (effective shrinking of the system under renor- 
malization) and it is a specific quantity hrei = Hrei/N, 
where N is e.g. a number of lattice sites, which is phys- 
ically relevant. A more careful analysis shows however 
that Hrei is much better that entropy H or similar quan- 
tities and possesses interesting monotonicity properties 
with respect to couplings (see Refs. [HI, for details). 
For example, close to a critical point with a diverging 
dimensionless correlation length ^ the relative entropy 
taken with respect to the fixed point distribution in many 
cases behaves as ~ iV/^'^ (for d-dimensional lattice sys- 
tem) and is universal, depending not on the total num- 
ber of variables, but only on the number of blocks of size 
^. This means that relative entropy does not contain 
redundant "low-level" information. When this is true 
hrei ^ 1/C is obviously monotonic along the Wilsonian 
RG flow, which is a particular example of how the infor- 
mational "distance" grows on a trajectory moving away 
from the unstable fixed point. 

If we take v in (jlip to be a uniform distribution ly = 
const, then Hrei = — In — _ff , where H is the entropy 
of a given distribution fi. Therefore, if the first term In v 
remains constant, the decrease of the relative entropy 
under a coarse-graining is usually related to the growth 
of the entropy H [3]. In our case, however, the first 
term also changes, because In i/ ~ iV due to normalization 
condition for v and therefore Hrei and H both decrease. 



III. MUTUAL INFORMATION OF FAST AND 
SLOW VARIABLES 

We now introduce another information-related quan- 
tity, the so called mutual information of the fast and slow 
variables, which seems to be free of the aforementioned 
drawbacks. We shall demonstrate now that the nonnega- 
tivity of this mutual information may itself impose some 
restriction on the RG flow. 

To define this mutual information, we need to perform 
a transformation opposite, in a sense, to the usual RG 
transform of Eq. ([2|), namely to integrate out the slow 
variables. This will result in an effective action S' for 
the fast variables tp, their probability distribution being 
p{ip) ~ exp(— 5"(V')), and the corresponding entropy will 
be denoted by H^ip). Then the mutual information of 
two sets of variables ip and rj is defined as 

/^5:,(,,^)inr^(^'^) 



pir])p{tp) 



(14) 



where pirj, tp) — 1/Zexp(— 5) is the probability distribu- 
tion for the initial system (joint probability for rj and ip 
variables). 

The mutual information is certainly a kind of rela- 
tive entropy which in our case measures the "distance" 



D(p{'ip,ri)\\p('ip)p{ri)) between the original system and 
some hypothetic one where fast and slow variables are 
independent, but described by their true corresponding 
effective actions, i.e. fi = p{tp, rf) and v — p{4')pi'n) ii^ 
Eq. (HI]). 

Substituting expressions for probabilities in terms of 
actions one can easily derive the well known formulas 
from elementary information theory 



/ = H{t]) - H{f]\tP) Hi^P) - H{^j\t]) = 
^[Hir,) + H{^j)]-Hii^,i^) 



(15) 



Here H{ip\ri) — SI is the information loss after the nor- 
mal RG transformation and H{ri\Tp) is the information 
loss after elimination of the slow variables, or if we adopt 
the view that renormalization is the information trans- 
mission through the noisy channel ^ t], then H{r]\'ip) 
is a measure of the noise [l^ . 

In bipartite systems the mutual information is nonneg- 
ative and is known to measure the information about one 
subsystem stored in the other [l6j (for independent sub- 
systems / = 0), i.e. in our case it shows what amount 
of information about the fast fields ^ is still contained in 
the effective action S'{ri). Since usually RG transforma- 
tion results in the change of coupling constants it seems 
natural to think that the information about the elimi- 
nated fast variables is now hidden in the new values of 
these constants. Therefore / should be directly related 
to the RG flow of the coupling constants. 

Note also that / is represented in the last equation of 
Eq. (jlSp as a difference of two quantities with roughly 
the same size dependence, so that it should not be so 
sensitive to the reduction of degrees of freedom under 
the RG transformation. 

While in general / is more difficult to calculate than 
the entropy, there exists an interesting situation when 
H{^/j) = H{ri) = H' and I = 2H' - H. This happens e.g. 
for some decimation transformations in lattice systems, 
when the system is divided into two identical sub-lattices 
(see next Section) and exactly half of variables (those 
on one sub-lattice) are integrated out. For large enough 
system with sites we may write H = Nh and H' = 
{N/2)h' where h and h' are the entropies per lattice site 
before and after the renormalization. Then 



/ = N{h' -h)>0 



h' > h 



(16) 



and hence the condition / > directly leads to the 
growth of the entropy per lattice site along the RG tra- 
jectory. 

This means that such decimations are always irre- 
versible, but suggests that corresponding RG flows do 
not have fixed points for finite generic values of cou- 
plings. Indeed, if there exists a fixed point then at this 
point h = h' and hence the bulk mutual information / 
is zero. But this means that spins on two sub-lattices 
are essentially uncorrelated and remain uncorrelated un- 
der renormalization, which seems impossible for realistic 
spin systems with reasonable finite interactions. Note, 
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that this does not imply that decimation does not have 
nontrivial fixed points at all, but such fixed points (apart 
from the trivial high-temperature one) should probably 
lie at the "boundary" of the parameter space, when some 
couplings are set to infinity, as one can see e.g. in the 
classical spin-1 model on a line [13] (see also section 4.2). 

The absence of a nontrivial fixed point corresponding 
to the phase transition for pure decimation in 2D Ising 
model have been noticed already in [l], [l8| and was re- 
lated to the fact that decimation does not eliminates only 
"short range" correlations but inevitably sums over some 
long-range behavior [l^ , i.e. it is actually not a good RG 
transformation. 

Let us now take a look at some simple examples. 

IV. EXAMPLES OF THE INFORMATIONAL 
APPROACH 

A. One-dimensional Ising model. 



where h{K) is the entropy per lattice site 

h{K) ^ -Kt&nhK + ln{2 cosh K), (18) 

and 

K' ^ (1/2) ln(cosh(2if)). (19) 

The RG fiow is very simple since the running coupling 
constant K fiows from the zero temperature fixed point 

= cxo to the high temperature one at i^T = 0. 

Though K' decreases during the RG procedure and 
h{K') > h{K) it is easy to verify that 51 is always pos- 
itive, has its maximum (A^/2)ln2 ai K = Q and then 
monotonically decreases to zero as X — > oo. 

It is interesting that for large K' the information 
loss can be obtained independently, so that Eq. ([T7|) can 
be used to derive K' . Indeed, \i K ^ oo most probable 
spin configurations are that of well separated kinks (do- 
main walls). If a spin, which is eliminated, falls inside 
a domain, no information is lost, because its value does 
not fluctuate at large K and is completely determined by 
adjacent spins. But a spin that lies on a domain wall is 
completely unconstrained and hence the information loss 
is In 2. Since there are on the average A^exp(— 2/^) kinks 
at if — > C)o we conclude that 




FIG. 1: Decimation transformations on (a) one-dimensional 
lattice and (b) two-dimensional square lattice. Spins on white 
lattice sites are summed away. 

The classical one-dimensional ferromagnetic Ising 
model is the most known example of the RG transfor- 
mation which here can be performed exactly. The model 
is defined on a lattice with N sites and the initial action 
is given by S* = — if X^i ^iSi+i, where Si = ±1. The real 
space RG procedure (decimation) is defined as follows: 
we decompose the lattice into two sub-lattices and sum 
over spins on one sublattice [11] (see Fig. la). The re- 
sulting model is again of the Ising type with K K', 
with twice the initial lattice spacing and N/2 sites. The 
entropy H is here the true entropy of the Ising model and 
the information loss, defined by Eq. ([7]), is 

N 

SI = Nh{K) - -hiK'), (17) 



SI ~Nexp{~2K)\n2 + (20) 

where dots indicate terms of higher order in exp(— 2if ). 
At large K the leading term in entropy is h{K) ~ 
{2K + 1) cxp{-2K) hence Eq. ^ results in 

{2K + 1- In 2) exp(-2is:) = {K' + ^) cxp{-2K'), (21) 

which has the solution 

K' = K -^\n2 + .... (22) 

This is the correct expression for the renormalized cou- 
pling at if — cx) and we see now that In 2 here is in fact 
of informational origin. 

Now we discuss the relative entropy Eq. ([TT|) of the 
Ising model at some coupling ifi taken with respect to 
the state at a different coupling if 2. The general formula, 
derived in [TT| states that 

Hrei ^ VF(ifi) - W{K2) - (ifi - ifa) , (23) 

OKi 

where 

W=-\iiZ. (24) 
In our case W^(if) = — iVln(2coshif) and hence 

Hrei{Ki,K2) = -N\n ^^^J^ + iV(ifi - if 2) tanhifi. 
cosh A 2 

(25) 
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We can look now at some limiting cases. First consider 
the entropy relative to the high temperature fixed point 
K2 = 0. Then 

Hrei{K,Q) = N{\n2~ h{K)) =N\n2-H. (26) 

One can easily see that Hj.ei decreases under the RG 
transformation when K K' and N — > N /2 and hrei = 
Hrei/N also tends to zero. 

The entropy relative to the zero temperature fixed 
point should be obtained in the limit K2 — > 00 but the 
limiting expression diverges. Fortunately, we can take 
Ki — 00 which is almost the same for our purposes. Then 



Hrei{oo,K) = N[-K + \n{2 cosh K)]. 



(27) 



Again the total relative entropy monotonically decreases 
along the RG flow as the general proof suggests, but now 
hrei increases from zero to In 2. Close to the critical point 

Hrei {oo,K)~ N exp{-2K) = 2N/^, K 00 (28) 

where ^ ~ l/2exp{2K) is the correlation length. Thus, in 
this limit Hrei is not changed by the RG transformation 
^ — ^/2, N — N/2 and actually measures the number of 
blocks of size ^ as the general analysis suggests [13 • 

This example clearly illustrates that Hrei always de- 
creases, but this behavior is in part due to the effective 
shrinking of the system N — > N/2 and implies only that 
hrei{K') < 2hrei{K). Thc relative entropy per lattice 
site hrei is more useful because it really measures the 
"distance" from thc fixed points and its evolution is re- 
lated to the direction of the flow. Since here the RG 
flow is from the zero temperature fixed point to the high 
temperature one, hrei{K,Q) decreases while hrei{oo, K) 
monotonically grows (as 1/^ close to the critical point). 

It is easy also to calculate the mutual information / 
for this model. Since the two sublattices are identical 
and similar to the original one, H{'tl>) = H{ri) = H' — 
{N/2)h{K') and hence 



I = 2H' -H = N[h{K') - h{K)] 



(29) 



We see then, that indeed /, as expected, is determined 
entirely by the change of the coupling constant and from 
/ > it follows that 



of Ising spins is well known Z = 2(2coshX)^^^. From 
this expression it is easy to obtain the entropy and its 
expansion in powers of 1/iV 



H{N) ^ Nh + C + 
where h is given by Eq. (|18p and 

C = KteinhK - In(coshX). 



(31) 



(32) 



This C monotonically decreases from ln2 at X = oo to 
zero at K = 0. It is interesting, that at fixed points 
C is equal to universal values Ing, where q is the de- 
generacy of the largest eigenvalue of the transfer matrix 
Tg^si = exp(ifss'), s,s' = ±1. Indeed, g = 2 at zero 
temperature fixed point (due to two ground states with 
opposite magnetization) while g = 1 in the fully disor- 
dered state. 

This subextensive part of the entropy is also known as 
the excess entropy and was studied in detail as a possible 
measure of statistical complexity of spatial structures in 
ID spin systems [ll|. It can be shown that if we divide 
our finite system into two halves then excess entropy C is 
equal to the mutual information of these halves (see [15 1 
and references therein). But any mutual information is 
a kind of relative entropy and so it should in general de- 
crease under coarse grainin g (if it preserves the partition 
of the block in two parts) [16| (see also Eq. (1121) ) and 
since it does not depend on N this decrease is not re- 
lated to the change of size N — >■ N/2. Thus for ID Ising 
model the excess entropy C has an independent meaning 
and its monotonic decrease under renormalization can be 
derived directly from thc information theory. 



B. Classical spin models with s > 1 in one 
dimension 



More interesting critical behavior may be obtained in 
one dimension if we take a spin chain with larger value 
of spin s. We start with the most studied example of a 
spin-1 chain which is defined by the action 



S = -Ki SiSi+i 



(33) 



h{K') > h{K), 



(30) 



i.e. the entropy per site h{K) always grows along the RG 
trajectory. This is rather trivial for the model in question 
since the RG fiow leads us to the disordered fixed point 
K = with maximum entropy regardless of the initial 
state because there is no phase transition. Nevertheless 
this simple example clearly shows how the irreversibility 
of the RG flow can be derived within thc informational 
approach. 

So far we have considered only the thermodynamic 
limit iV — > (X). Let us now take a more careful look at 
the flnite system. The partition function for finite block 



where spin now takes three possible values Si = 0,±1. 
This is again a ferromagnetic model for Ki > 0, but 
with much richer behavior than before because we can 
now vary the effective length of the spin by changing 
the parameter K^. The action p3|) retains its form un- 
der the decimation transformation and only the coupling 
constants are changed. 

It is widely believed that there are no phase transi- 
tions in one-dimensional systems with short range inter- 
actions. This is true, however, only if all the couplings 
are finite. Non-trivial critical behavior in this model, an- 
alyzed thoroughly in Ref. [l3|, takes place when both 
K2 and K3 tend to infinity while K2 — remains finite. 
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The remaining two-dimensional space of couplings may 
be parameterized by new coordinates 

X = exp(-ifi - ^2 + A'3), y ^ exp(-2i^i) (34) 

and the decimation transformation acts as 

2y 



1 + 



y 



1 + 



(35) 







h=ln2 




h=y2ln2 






/* C 










X 


2 



FIG. 2: Decimation RG flow for the spin-1 model [l7| with 
the values of entropy per lattice site h at some fixed points. 

The resulting quite interesting RG flow diagram ob- 
tained in is shown schematically in Fig. 2. Its 
main features are four fixed points and a critical line AC. 
Since this flow arises from the decimation transformation 
our general arguments require that the entropy per site 
h{x, y) should increase alon g th e flow. Though the model 
can be solved exactly (see [17| for details) it is easy to 
evaluate the entropy at the fixed points without much 
computations. 

The point A corresponds to fully ordered state (actu- 
ally there are three such states with all Si either -1-1, or 
— 1, or 0) hence Ha = H/N at — > cx) with only 
subextensive contribution to H. The same is true also 
for the point x = y = with two ordered states. The 
point B is similar to a high temperature fixed point of 
the Ising model, with uncorrelated Si — ±1 and no zero 
spins, hence hs = In 2. 

The most interesting case is the critical point C at a; = 
2,y = 1. It corresponds to i^i = while K3 — K2 = ln2. 
In this case there exist the lowest energy state with all 
Si — and zero energy. For periodic boundary condi- 
tion all configurations with some finite fraction of spins 
changed to ±1 have infinite energy at Kg, 00 while 
states with all Si = ±1 are gapped with AE — A^ln2. 
But there are 2^ = exp{N In 2) such states and their sta- 
tistical weight exactly compensates the Boltzmann factor 
exp(— A£^) which results in the partition function Z = 2 
and mean energy (S) — AE/2. Therefore at C we effec- 
tively have two equally probable states with highly de- 
generate disordered excited one and the resulting entropy 
he = {{S) +lnZ)/N (1/2) ln2 at A^ ^ 00. 

The values of entropies are shown in Fig. 2 near the 
corresponding fixed points. We see now that the entropy 



per site can in fact be used to understand the ordering 
of the fixed points along the flow since h monotonically 
grows when we pass from A to C and finally to B. 

Certainly this is not the only way to classify these 
fixed points. In fact we can use also the degeneracy q of 
the largest eigenvalue of the transfer matrix T (actually 
the degeneracy of the ground state of the correspond- 
ing quantum Hamiltonian H, defined by T = exp(— iJ)). 
In ID g plays, in a sense, the same role as the central 
charge c in two dimensions, because q always decreases 
under renormalization (this also resembles the (/-theorem 
(20} for ID quantum systems). The global decrease of q in 
ID spin system with nearest-neighbor interactions when 



S 



'V{s,,s,+i) 



(36) 



is rather trivial and follows directly from the decimation 
RG transformation written as a transfer matrix mapping 



rp ^ rj-i/ rj-i2 



exp{-V{s,s')) 



(37) 



where for spin with n possible values T is a nonnegative 
nxn matrix. Eigenvalues of T transform as Ai — >■ = Af 
so that the degeneracy once reduced will never return 
back. For strictly positive T Frobenius theorem requires 
q = 1, but when some couplings turn to infinity, as we 
have seen above for the spin-1 model, some entries in T 
may vanish and q may differ from unity. For the spin-1 
model all these eigenvalues were calculated in Ref. ^3] 
and q = 3 for point A, q — 2 for C (also for the point at 
the origin) and q — 1 for point B in Fig. 2. 

For these models it is possible also to evaluate the ex- 
cess entropy at the fixed points. Fixed points for deci- 
mation are defined by 



rj-if rj-i'2 



XT 



(38) 



where A is some positive number (multiplication of T 
by a constant does not change the physics) and it can 
be easily shown that in these cases T may have q < n 
eigenvalues A while other eigenvalues are zeros. Then for 
the periodic boundary condition 



Z = tr(r 



qX 



N 



In Z = A^ In A -I- In g (39) 



and since (S) ^ N entropy at fixed points has the form 
Nh + C with C — \nq. In this case the subextensive part 
in H coincides with that in In Z and this number mono- 
tonically decreases when we pass from one fixed point to 
another along the RG flow (this is somewhat similar in 
spirit to the recently proposed i^-theorem for field theo- 
ries in odd dimensions [5]). Unfortunately, for periodic 
boundary conditions C equals In q also away from fixed 
points (at N — 00) and it is discontinuous, because q 
changes abruptly, while excess entropy for a finite block 
in an infinite chain, which is related to mutual informa- 
tion of two halves and is expected to be continuous and 
monotonic along the RG fiow, is much harder to evaluate 
at s > 1 and in general is no more related to g in a simple 
way even at fixed points. 
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C. Decimation in two dimensions. 

Consider now a two-dimensional classical spin system, 
with spins placed on sites of a square lattice. Various spin 
interactions, including e.g. nearest-neighbor, second- 
nearest-neighbor and similar interactions, are parameter- 
ized by a set of coupling constants K = {Ki, K2, ■ ■ ■)■ To 
define a "checkerboard" decimation transformation we 
first divide the lattice into two interpenetrating square 
sub-lattices and then sum over all spins on one sublat- 
tice (see Fig. lb) . Just as in one dimension we obtain 
then a new spin system again on a square lattice (with 
the lattice spacing \/2 times larger) but with a new en- 
ergy S' with couplings K'. This is the first step of the 
RG transformation, which is then repeated further. 

Since two sub-lattices are identical the mutual infor- 
mation between them is again given by the Eq. (P^ . 
i.e. 

I/N = h{K') - h{K) > (40) 

for a large lattice with TV sites, where ft.(K) is the en- 
tropy per site. Hence we conclude that in this case the 
entropy per site should also increase in the course of the 
RG transformation and the RG flow is irreversible. 

The entropy growth in two dimensions may seem a 
rather counterintuitive result, since in systems with phase 
transition one may naively expect that at low temper- 
atures (below the transition) the RG transformation 
should lead us deeply into the ordered phase with /i(K') 
decreasing to zero. But for the decimation this intuition 
obviously doesn't work, since in this case the average spin 
remains unchanged so that the spontaneous magnetiza- 
tion is constant along the RG flow. 

For better understanding of why the entropy per site 
can grow under renormalization at low temperatures let 
us now consider the ferromagnetic Ising model with S — 
—Ki^^^SiSj, where the sum is over nearest-neighbors 
and Si = ±1. After the first step of the RG we have (up 
to an unimportant constant) [l[ 

4 

S' - -K[ J2 s^SJ - K'^ J2 ^^^^ - ^3 E n (41) 

nn nnn P i—1 

where the second term is the next-nearest-neighbor in- 
teraction and the last one is the four spin interaction (si, 
i = 1 ... 4 are spins on the corners of a plaquette) with 
the sum over all plaquettes. The new coupling constants 
are known to be 

K[ ^2K^^ \ ln(cosh4Xi), (42) 
if^ = I ln(cosh4ii:i) - i ln(cosh2ii:i) (43) 

We see now that K[ > (though K[ < Ki), the next- 
nearest-neighbor interaction is also ferromagnetic, K2 > 
0, but K'^ < 0, i.e. the four-spin interaction tries to 
decrease the ferromagnetic order. 

Next, we estimate the entropy at low temperatures. At 
Ki — 00 all spins point in one direction and the entropy 



is different from zero only due to rare spin flips. The par- 
tition function may be represented as a low temperature 
expansion 

Z = 2e-'^^° (1 + Ne^p{-AE{K)) + ...), (44) 

where NEq is the energy of all spins aligned and the 
energy needed to reverse one spin is AE — 8K1 in original 
system since after the reversal four links have "wrong" 
alignment of adjacent spins. Hence at low temperatures 
the entropy per site is 

h ~ 8K1 exp{-8Ki) + ... (45) 

After the decimation the system is still ferromagnetic but 
the energy of spin reversal is now AE' = 8K[+8K2+8K'^ 
or 

AE' = 4 (ln(cosh4/s:i) - In (cosh 2/^1 )) ~ 8K1 - 4e"^^i 

(46) 

at large Ki. Hence the new excitation energy is smaller 
than AE and it becomes easier to reverse a spin. How- 
ever, since the difference is exponentially small one has 
to include also the next term in /i(K) due to pairs of re- 
versed spins 24Kiexp{—12Ki) (the corresponding term 
in /i(K') can be easily evaluated as ^ exp{—16Ki) and 
is unimportant). But even with this contribution taken 
into account the entropy per site after the decimation 
h{K') ~ A£:'exp(-A£:') is larger, 

h{K') - h{K) ~ 8i^ie-i2ifi > 0. (47) 

Thus, it is possible to check at least for the first itera- 
tion at low temperatures that indeed the entropy per site 
grows after the decimation. Physically the result (|47| for 
the mutual information or / ~ 7Vexp(— 12fsri) is due to 
the reversed pairs of adjacent spins in the original lattice. 
Only through these pairs the two sub-lattices have non- 
trivial information about each other at Ki — 00 (there 
should be also a term In 2 in / since the orientations of 
the overall magnetization in sub-lattices are correlated, 
but we are interested only in terms ~ A'^ in the mutual 
information). 

Now it is possible to explain, what this increase in en- 
tropy at low temperatures actually means. Since the av- 
erage magnetization conserves along the decimation RG 
flow, the average number of reversed spins per unit vol- 
ume remains the same, but now more of them are isolated 
ones (with less energy cost of reversal) and the number of 
reversed close pairs or larger clusters is relatively smaller 
than in the initial system. Surely this picture may also 
be interpreted as a decrease in the correlation length for 
spin fluctuations. 

Unfortunately the next iteration of the decimation can- 
not be performed exactly and is known to lead to non- 
Gibbsian measures at low temperatures [211 1, so that it 
is not possible to describe renormalization as a flow in 
usual space of couplings. Still there should be a flow of 
probability measures, probably with limiting points cor- 
responding to product measures describing independent 
spins with fixed total magnetization. 
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D. KadanofF block-spin transformation 

Consider now a classical lattice system of Ising spins 
Si with the action So[s] and first divide the lattice into a 
set of nonoverlapping identical blocks with, say, k adja- 
cent spins in each block Bj, assuming that blocks form 
a lattice, similar to the original one (see Fig. 3 for an 
example). Then, following KadanofF prescription [2^ as- 
sociate a new spin variable tj with j-th block and define 
the RG transformation 5*0 [s] — > S' [t] according to 

exp{S'[t]) = J2 Tit, s) exp(-5oW) (48) 

S 

with T(t, 5) > obeying the condition 

Y.T{t,s) = \ (49) 
t 

which guarantees that the partition function Z is the 
same for both s and t variables. 




FIG. 3: Block-spin transformation on a triangular lattice for 
= 3. Blocks Bj are shaded, spins Si are located at black 
sites while block variables tj may be placed in the centers of 
blocks. 

This transformation is of the general type of Eq. ^ 
but now all Si are "fast" variables and tj are "slow" ones. 
Hence the total "action" S\t, s] = — lnT(i, s) + So[s\ now 
defines the joint probability for initial and "renormal- 
ized" spins. The entropy H now is not equal to the 
physical entropy Hq of the initial spin system but is given 
by 

H{t,s) = {S)s + \nZ={S(,)s, - (lnr)5 + lnZ^50) 
= Ho-(lnT)s, 

where we have used the condition in the calculation 
of (5o)s. 

The most popular form of T(t, s) is 



where p is a free parameter and 

is the sum of initial spins in the block Bj. Note that 
ptj enters in the partition function as a spatially varying 
magnetic field constant over a block. Hence at p — > cxd 
all block spins Sj will be aligned strictly along tj. Now 
making use of the identity {t'j)s = {^^'^^{pSj))so eas- 
ily obtain 

-(lnr)s = E,(/b^.-)>5„, (53) 
f{x) — — xtanhx -|- ln(2cosha;), 

where the sum is over all blocks. Next, we can easily sum 
exp(— S'[i, s\) over t because of the identity (|49|) to obtain 
S'[s\ — Sq[s\ and hence 

H{s) - i/o (54) 

Now we can use the definition of the mutual informa- 
tion (fT5|) and Eqs. ([50|l . (j54|) to derive the final expression 

/ = Hit) + His) - Hit, s) = Hit) -Y,{fipSj))s, (55) 

j 

Note that Hq cancels away from the resulting expression 
for the mutual information. Next we introduce the en- 
tropy per lattice site for renormalized spins according to 
Hit) = (7V//c)/i(K') and instead of / = iV(/i(K') - /i(K)) 
which was valid for decimation, we have at large N quite 
a different expression 

/=^[MK')-(/(p$]s.))o]>0, (56) 

i 

where the sum of spins Si is over one block, k is the 
number of spins in a block and the average is over the 
initial configurations with the weight exp(— S'o[s]). The 
second term in Eq. ([55]) looks like the entropy of the 
total spin of a block at temperature l/p averaged over 
different values of its magnitude. 

Thus the entropy after the renormalization /i(K') is 
always bounded from below, provided the second term 
in Eq. (j56p is nonzero. This is due to the noise induced 
by the renormalization at finite p. 

The most interesting case however corresponds to p — >■ 
00 which is known to lead to the so called majority-rule 
renormalization (2^. This means that we define tj as ±1 
depending on the sign of the total block spin Sj. We 
assume also that k is odd to avoid ambiguities. Then 
Sj ^ and from Eq. ^ it follows that fipSj) ^- at 
p — !■ 00 leading to 

N 

I = H' = —hiK'), p~^oo, fc-odd. (57) 
k 



This result is rather obvious however, because from Eq. 
(ITSI) we have / = H' — i?(i|s) but for majority-rule RG 
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the initial configuration of spins Si completely determines 
that of renormahzed spins tj so that H{t\s) = (there is 
no noise in the transmission channel) and hence their mu- 
tual information coincides with the final entropy I = H' . 
Therefore in this case the positivity of mutual informa- 
tion gives nothing new and ^(K') can be either greater 
or smaller than /i(K). Contrary to decimation now / > 
does not prohibit fixed points and the entropy per site 
can even go to zero along the RG flow. 



E. One dimensional free field theory 

Now we proceed to quantum field theory, using ordi- 
nary quantum mechanics as the simplest example. Con- 
sider a quantum particle of mass m, moving along a line 
in a harmonic potential well with a frequency a;. Its 
partition function at a temperature T is given by a path 
integral, which in a lattice regularization, when the imag- 
inary time interval [0,(3] {f3 = l/T) is divided into N 
intervals of length e, has the form 



V27re 



N/2 



/ exp 

s-l^T : + ^2.^^- 



(58) 



with A'^ — /3/e. Shannon entropy, as mentioned above, 
equals 



H={S)+ In Z, 



(59) 



where brackets mean the average with weight exp(— S*). 
H coincides with the entropy of the statistical system of 
variables Xi placed on sites of a one-dimensional lattice of 
length /3 with nearest-neighbor interaction and periodic 
boundary condition. This system is actually the one- 
dimensional Gaussian model near its critical point uj — 

M- 

However, at e ^ this model is actually (0-1-1) Eu- 
clidean free field theory and its thermodynamic entropy 
is given by a different formula 



Ht = {E - F) /T = EP + lnZ 



(60) 



where E is the mean energy of the particle and F = 
— TlnZ is its free energy. Let us now briefly discuss the 
connection between these two entropies. 

The mean velocity squared (i^) which enters in (S) 
diverges as e — >■ j25| and its direct evaluation is not so 
easy, but the average action can be evaluated simply by 
differentiating Z with respect to mass 



(61) 



where the first term in the r.h.s. comes from the dif- 
ferentiation of m in the integration measure. But for 



the harmonic oscillator Z does not depend on mass, and 
therefore (S) — N/2. In the continuum limit 



InZ 



-ln(2sinh(a;/3/2)), 



(62) 



and at a given uj we have the following asymptotics for 
the entropies at /? — oo 



Ht - 



oj/3e 



(63) 
(64) 



We see that the leading "bulk" term in H is divergent in 
the continuum limit e and Ht resembles, in a sense, 
the finite size correction in H . 

The different behavior of H and Ht comes from the 
fact that Ht describes the information related to energy 
levels occupation and hence tends to zero at zero tem- 
perature when the system is in its ground state, while H 
describes fiuctuations of "paths" in configuration space 
which obviously grow as /3 — ?> oo. 

The entropy per lattice site here is 



1 euj 
h = 

2 2 



(65) 



at e — > 0. It may seem strange that h decreases with 
correlation length ^ = (we)"^, because for the model in 
question we may also apply the decimation transforma- 
tion, similar to that in the Ising model, integrating away 
XiS on alternating sites (see e.g. [1^) and our general 
result states that in this case h should increase along the 
RG flow. Under such a decimation ^ — ^/2 and h from 
Eq. (1651) does not behave in the required way. 

The solution to this apparent contradiction is rather 
simple. Note first that summation over variables x is 
defined now as 



(66) 



and the integration measure itself depends on mass and 
on lattice spacing. This integration measure is fine-tuned 
to the action so that Z has a well defined correct con- 
tinuum limit ([5^ . But after the RG transformation this 
fine-tuning is destroyed. The effective action S' now con- 
tains renormahzed mass m', frequency w' and a new lat- 
tice spacing 2e, but the measure (|66p remains the same. 
This means that in this case the entropy after renormal- 
ization is no longer given by the same formula i6S\) as 
before. 

We may transform the measure to the correct form but 
this leads to an additional factor (2m /m')^^^ in Z, since 
the lattice now has N/2 sites. This additional factor does 
not affect the mean action (S") but certainly change the 
In Z part of the entropy. Simple Gaussian integration 
like that in Ref. [26^] shows that m' ~ m up to the terms 
of order at e — >■ and the additional factor is simply 
2-/v/4^ Then the new entropy per site equals 



h' 



1 



■ln2-^ 



(67) 
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at e — )■ 0. This quantity is larger than /i ~ 1/2 and the 
mutual information / of two sublattices is given by 



N 

I = N{h' -h) = y ln2 



(68) 



at e — > 0. Note also that the total entropy is reduced 
after the decimation, because the information loss 51 — 
H - H' = Nil- ln2)/4 is positive. 

Let us now analyze this system more carefully. After 
the decimation is repeated many times we get away from 
the continuum limit because the lattice spacing grows. 
In this region it is more convenient to rewrite the renor- 
malized action in the form 



(69) 



where K , b are new running coupling constants with the 
initial values = m/e, bo = 2m/e + emu!'^. The entropy 
per lattice site for this action can be derived from the 
exact solution of the Gaussian model [2JI 



h = 



11 fKo\ 1 
2 + 2^n-)-2 



dt / K 
— In 1 - 2— cost = 



27r 
1 



VI - 4z2 



where Kq = m/e comes from the integration measure in 
Eq. Note that while all correlations depend only on 

z = K/b, the entropy depends on both K and b because 
the integration measure contains the unrenormalized cou- 
pling Kq. Decimation acts as 



K' 



b ' 



b' ^b-2 



(71) 



or z' = z^jiX — 2z'^). This RG equation has a fixed 
point z* — 1/2 which corresponds to a; = and under 
renormalization z flows to z = 0. Close to the fixed point 
z = 1/2 one can define a continuous theory by choosing 
if ~ 6/2 ~ m/e and b-2K = emuj^ with e ^ and RG 
equations turn into K' ~ K/2, b' ~ 6/2. 

It is now easy to verify that in the vicinity oi z — 
1/2 the initial value of the entropy is /i = 1/2 and that 
it indeed acquires an additional (1/2) In 2 contribution 
at each decimation step provided we are still close to 
the fixed point. Note that h grows even for z — 1/2 
because in the two dimensional space of K and 6 there is 
no critical fixed point and the point z = 1/2 corresponds 
to a line K = 6/2 with the RG flow along it. After many 
decimations z — > 0, i.e. K decreases much faster than 6 
and the entropy starts to behave as 



1 



■In 



6 



if < 6 < i^o 



(72) 



with logarithmic accuracy. Since if — >■ means the loss 
of correlations between x's on different sites, this expres- 
sion for the entropy h is in fact the entropy production 
rate for the Gaussian white noise /i ~ In cr with dispersion 
~ 1/6, well known in the information theory. 



V. CONCLUSION 

In this paper we have tried to understand what lim- 
itations on the RG flow may follow from the informa- 
tion theory, because Wilsonian RG transformation, when 
some fast variables are integrated out, is obviously re- 
lated to the information loss. This transformation is sim- 
ilar in a sense to a signal transmission through a noisy 
channel, the fast variables being the input, while the slow 
ones play the role of the output available to a receiver. 
The information loss is defined here as an average condi- 
tional entropy of the eliminated fast variables when the 
remaining slow variables are fixed. From the nonnega- 
tivity of this quantity (which holds at least for systems 
with discrete variables, like e.g. classical spins) it fol- 
lows immediately that the informational entropy H of 
the system cannot increase under renormalization. 

Unfortunately this result is not very interesting, and 
does not lead to the irreversibility of the RG flow, be- 
cause it applies to the total extensive entropy, propor- 
tional to the size of the system, which effectively shrinks 
after renormalization. This shrinking is evident for real 
space RG schemes, where some lattice sites are elimi- 
nated and the system size is smaller if measured in new 
lattice constant, but equally applies to momentum space 
Wilsonian renormalization due to final rescaling of mo- 
menta. 

Only if H possesses some volume independent additive 
part C, then this part, if it also decreases under renor- 
malization, may provide the required Lyapunov function, 
that depends only on couplings. Physically its decrease 
may be attributed to a loss of information about mass- 
less modes, but this does not follow directly from that 
of H and should be deduced independently. Then, to be 
monotonous C must have also some independent defini- 
tion. For ID Ising model this quantity is just the subex- 
tensive excess entropy and we explicitly check its mono- 
tonicity under RG transformation. This behavior for a 
finite block of spins in an infinite chain follows directly 
from information theory, because excess entropy in this 
case is just the mutual information of two halves of the 
system (isj . which decreases under coarse graining. 

Note also that for ID systems with periodic bound- 
ary conditions the excess entropy is Ing where q is the 
degeneracy of the largest eigenvalue of the transfer ma- 
trix which always decreases along the RG flow. This is 
somewhat similar to what is known for critical 2D sys- 
tems defined on a long cylinder of size x Ly with, say, 
Ly — >■ cx) and periodic boundary condition in the x di- 
rection. In this case the first finite size correction in the 
expansion of entropy in l/L^; is ~ c{Ly/Lx) [Ml, where 
c is the central charge. Therefore the subextensive part 
C in entropy in 2D should decrease as a consequence of 
the c-theorem. However in both these examples the sub- 
leading term in entropy is discontinuous and does not 
directly lead to a proper Lyapunov function. 

This means that the extraction of the required part 
from the entropy of finite size system is somewhat am- 
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biguous and depends on the boundary condition. One 
may suggest, however, that in general there might ex- 
ist some size-independent information-based quantity de- 
fined on the whole space of couplings (maybe some kind 
of relative entropy or higher order mutual information 
[27I I) that is monotonic along the flow and at fixed points 
coincides with some kind of excess entropy, but this point 
is not clear. 

The main aim of the present paper was, however, apart 
from demonstrating some examples of informational ap- 
proach, to introduce a different quantity, namely the 
mutual information of fast and slow variables /, which 
shows how much information about the eliminated vari- 
ables is still stored in the renormalized effective action. 
This quantity does not suffer from the "shrinking size" 
trouble and is less sensitive to the overall decrease in the 
number of degrees of freedom, though it is generally more 
difficult to evaluate. 

There exist however some real space decimation trans- 
formations for which this mutual information can be eas- 
ily calculated. This happens e.g. if the original lattice 
may be decomposed in two identical sublattices and RG 



transformation eliminates spins on one sublattice. Then 
the nonnegativity of / results in the increase of entropy 
per lattice cite and RG flow resembles a relaxation to 
equilibrium. This result does not depend on interactions 
and may be true also for a larger set of models. What is 
actually needed is that the RG transformation must be 
represented as a product of such decimations. 

The approach to irreversibility based on the mutual 
information is different from those trying to general- 
ize Zamolodchikov's c-theorem and possibly may provide 
some additional information about the RG flows. It may 
appear interesting also to apply the present approach to 
the momentum space renormalization in fleld theory or 
to study more complicated correlations along the flow 
and some higher-order mutual information. 
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